Model-free Dense Stereo Reconstruction Creating Realistic 3D City Models

In this paper we describe a framework for fully automatic and model-free generation of accurate and realistic 3D city models using multiple overlapping aerial images. The underlying DSM is computed by dense image matching, using a robustified Census transform as cost function. To further reduce the noise of mismatches, we afterwards minimize a global energy functional incorporating local smoothness constraints using variational methods. Due to the convexity of the framed problem, the solution is guaranteed to converge towards the global energy minimum and can be efficiently implemented on GPU using primal-dual algorithms. The resulting point cloud is then being triangulated, local planarity constraints are exploited to reduce the number of vertices and finally a multi-view texturing is applied. The quality of the DSM and the 3D Model is evaluated on a complex urban environment, using reference data generated by laser scanning (LiDAR)

SceneFlowFields: Dense Interpolation of Sparse Scene Flow Correspondences

While most scene flow methods use either variational optimization or a strong rigid motion assumption, we show for the first time that scene flow can also be estimated by dense interpolation of sparse matches. To this end, we find sparse matches across two stereo image pairs that are detected without any prior regularization and perform dense interpolation preserving geometric and motion boundaries by using edge information. A few iterations of variational energy minimization are performed to refine our results, which are thoroughly evaluated on the KITTI benchmark and additionally compared to state-of-the-art on MPI Sintel. For application in an automotive context, we further show that an optional ego-motion model helps to boost performance and blends smoothly into our approach to produce a segmentation of the scene into static and dynamic parts.Comment: IEEE Winter Conference on Applications of Computer Vision (WACV), 201

Airborne Crowd Density Estimation

This paper proposes a new method for estimating human crowd densities from aerial imagery. Applications benefiting from an accurate crowd monitoring system are mainly found in the security sector. Normally crowd density estimation is done through in-situ camera systems mounted on high locations although this is not appropriate in case of very large crowds with thousands of people. Using airborne camera systems in these scenarios is a new research topic. Our method uses a preliminary filtering of the whole image space by suitable and fast interest point detection resulting in a number of image regions, possibly containing human crowds. Validation of these candidates is done by transforming the corresponding image patches into a low-dimensional and discriminative feature space and classifying the results using a support vector machine (SVM). The feature space is spanned by texture features computed by applying a Gabor filter bank with varying scale and orientation to the image patches. For evaluation, we use 5 different image datasets acquired by the 3K+ aerial camera system of the German Aerospace Center during real mass events like concerts or football games. To evaluate the robustness and generality of our method, these datasets are taken from different flight heights between 800 m and 1500 m above ground (keeping a fixed focal length) and varying daylight and shadow conditions. The results of our crowd density estimation are evaluated against a reference data set obtained by manually labeling tens of thousands individual persons in the corresponding datasets and show that our method is able to estimate human crowd densities in challenging realistic scenarios

Efficient Large-Scale Stereo Reconstruction using Variational Methods

This thesis investigates the use of convex variational methods for depth reconstruction from optical imagery and fusion of multiple depth maps into combined depth maps with higher accuracy. Dense depth reconstruction from two or more camera views are an important subject of research in computer vision, since measurement density is much higher than other depth sensing techniques, namely active depth sensing via infrared pattern projection or Lidar and Radar based techniques - even though the latter ones are more accurate and robust in depth. Other advantages of cameras are their low costs and low power consumption due to their passive sensing principle. Approaches are ranging from autonomous driving cars, obstacle avoidance or surveying UAVs up to detailed reconstruction of remote terrains using spaceborne imagery. In particular, we propose a fast algorithm for high-accuracy large-scale outdoor dense stereo reconstruction. To this end, we present a structure-adaptive second-order Total Generalized Variation (TGV) regularization which facilitates the emergence of planar structures by enhancing the discontinuities along building facades. Instead of solving the arising optimization problem by a coarse-to-fine approach, we propose a quadratic relaxation approach which is solved by an augmented Lagrangian method. This technique allows for capturing large displacements and fine structures simultaneously. For the application in autonomous driving, we further present an algorithm for dense and direct large-scale visual SLAM that runs in real-time on a commodity notebook. We developed a fast variational dense 3D reconstruction algorithm which robustly integrates data terms from multiple images thus enhancing quality of the Image matching. An additional property of this variational reconstruction framework is the ability to integrate sparse depth priors (e.g. from RGB-D sensors or LiDAR data) into the early stages of the visual depth reconstruction, leading to an implicit sensor fusion scheme for a variable number of heterogeneous depth sensors. Embedded into a keyframe-based SLAM framework, this results in a memory efficient representation of the scene and therefore (in combination with loop-closure detection and pose tracking via direct image alignment) enables us to densely reconstruct large scenes in real-time. Finally, applied to space-borne remote sensing, we present an algorithm for robustly fusing digital surface models (DSM) with different ground sampling distances and confidences, using explicit surface priors to obtain locally smooth surface models. The optimization using L1 based differences between the separate DSMs and incorporating local smoothness constraints is also inherently able to include weights for the input data, therefore allowing to easily integrate invalid areas, fuse multiresolution DSMs and to weigh the input data

LARGE SCALE URBAN RECONSTRUCTION FROM REMOTE SENSING IMAGERY

Automatic large-scale stereo reconstruction of urban areas is increasingly becoming a vital aspect for physical simulations as well as for rapid prototyping large scale 3D city models. In this paper we describe an easily reproducible workflow for obtaining an accurate and textured 3D model of the scene, with overlapping aerial images as input. Starting with the initial camera poses and their refinement via bundle adjustment, we create multiple heightmaps by dense stereo reconstruction and fuse them into one Digital Surface Model (DSM). This DSM is then triangulated, and to reduce the amount of data, mesh simplification methods are employed. The resulting 3D mesh is finally projected into each of the input images to obtain the best fitting texture for each triangle. As verification, we provide visual results as well as numerically evaluating the accuracy by comparing the resulting 3D model against ground truth generated by aerial laser scanning (LiDAR).

Large Scale Urban Reconstruction from Remote Sensing Imagery

Automatic large-scale stereo reconstruction of urban areas is increasingly becoming a vital aspect for physical simulations as well as for rapid prototyping large scale 3D city models. In this paper we describe an easily reproducible workflow for obtaining an accurate and textured 3D model of the scene, with overlapping aerial images as input. Starting with the initial camera poses and their refinement via bundle adjustment, we create multiple heightmaps by dense stereo reconstruction and fuse them into one Digital Surface Model (DSM). This DSM is then triangulated, and to reduce the amount of data, mesh simplification methods are employed. The resulting 3D mesh is finally projected into each of the input images to obtain the best fitting texture for each triangle. As verification, we provide visual results as well as numerically evaluating the accuracy by comparing the resulting 3D model against ground truth generated by aerial laser scanning (LiDAR)

3D Reconstruction Chain - From Images to 3D City Model

1. Fundamentals - Theory pinhole camera model - Camera calibration / undistortion - Pose estimation - Bundle adjustment 2. 3D - Sparse Reconstruction - n-View feature matching - Multi ray intersection 3. 3D - Dense Reconstruction - Epipolar Geometry / Fundamental matrix - Planesweep approach - Generation of Disparity Space Image - Image Matching - cost functions - Adaptive Support Windows - Cost aggregation / Regularization techniques (WTA, SGM, MRF, Total Variation) 4. Post-Processing - Consistency check (LR-check) - Outlier filter - Interpolation - Sub-disparity refinement - 2.5D depthmap fusion - 3D fusion 5. 3D Modeling - Meshing of point clouds / DSMs - Mesh simplification for data reduction - Multi-view texturin

Evaluation of Skybox Video and Still Image products

The SkySat-1 satellite lauched by Skybox Imaging on November 21 in 2013 opens a new chapter in civilian earth observation as it is the first civilian satellite to image a target in high definition panchromatic video for up to 90 seconds. The small satellite with a mass of 100 kg carries a telescope with 3 frame sensors. Two products are available: Panchromatic video with a resolution of around 1 meter and a frame size of 2560x1080 pixels at 30 frames per second. Additionally, the satellite can collect still imagery with a swath of 8 km in the panchromatic band, and multispectral images with 4 bands. Using super-resolution techniques, sub-meter accuracy is reached for the still imagery. The paper provides an overview of the satellite design and imaging products. The still imagery product consists of 3 stripes of frame images with a footprint of approximately 2.6 x 1.1 km. Using bundle block adjustment, the frames are registered, and their accuracy is evaluated. Image quality of the panchromatic, multispectral and pansharpened products are evaluated. The video product used in this evaluation consists of a 60 second gazing acquisition of Las Vegas. A DSM is generated by dense stereo matching. Multiple techniques such as pairwise matching or multi image matching are used and compared. As no ground truth height reference model is availble to the authors, comparisons on flat surface and compare differently matched DSMs are performed. Additionally, visual inspection of DSM and DSM profiles show a detailed reconstruction of small features and large skyscrapers

Fast and Accurate Large-scale Stereo Reconstruction using Variational Methods

This paper presents a fast algorithm for high-accuracy large-scale outdoor dense stereo reconstruction of manmade environments. To this end, we propose a structureadaptive second-order Total Generalized Variation (TGV) regularization which facilitates the emergence of planar structures by enhancing the discontinuities along building facades. As data term we use cost functions which are robust to illumination changes arising in real world scenarios. Instead of solving the arising optimization problem by a coarse-to-fine approach, we propose a quadratic relaxation approach which is solved by an augmented Lagrangian method. This technique allows for capturing large displacements and fine structures simultaneously. Experiments show that the proposed augmented Lagrangian formulation leads to a speedup by about a factor of 2. The brightness-adaptive second-order regularization produces sub-disparity accurate and piecewise planar solutions, favoring not only fronto-parallel, but also slanted planes aligned with brightness edges in the resulting disparity maps. The algorithm is evaluated and shown to produce consistently good results for various data sets (close range indoor, ground based outdoor, aerial imagery). 1